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Preface 


This  report  is  1  of  6  interdependent  research  outlines  in  the  Adaptive  Training 
research  program.  Portions  of  this  text,  which  originated  in  ARL-SR-0325,1 
appear  in  all  6  reports  to  ensure  that  readers  get  the  same  cross-cutting 
information. 


*  Sottilare  R,  Sinatra  A,  Boyce  M,  Graesser  A.  Domain  modeling  for  adaptive  training  and  education  in 
support  of  the  US  army  learning  model — research  outline.  Aberdeen  Proving  Ground  (MD):  Army  Research 
Laboratory  (US);  June  2015.  Report  No.:  ARL-SR-0325. 


VI 


1.  Introduction 


Training  and  education  tools  and  methods  must  be  of  sufficient  intelligence  to 
understand  the  needs  of  individual  learners  and  units  of  learners,  to  mitigate 
negative  learner  states,  and  to  guide  and  tailor  instruction  in  real  time  to  optimize 
learning.  These  tools  and  methods  must  also  be  affordable,  effective,  and  easy  to 
access  and  use.  These  requirements  are  enablers  of  the  US  Army  Learning  Model 
(ALM),  which  includes  an  emphasis  on  self-regulated  learning  (SRL)  where 
Soldiers  are  expected  to  manage  their  own  learning  and  career  development 
through  the  growth  of  metacognitive  (e.g.,  reflection),  self-assessment,  and 
motivational  skills  (Butler  and  Winne  1995).  While  SRL  skills  are  difficult  to 
train  and  develop,  support  may  be  provided  to  the  learner  through  “adaptive 
training  technologies”  (tools  and  methods),  which  may  be  focused  to  guide 
learning  and  reinforce  SRL  principles. 

To  support  ALM,  the  US  Army  Research  Laboratory  (ARL)  has  developed  a 
program  of  research  called  “adaptive  training”,  which  includes  6  interdependent 
research  areas  or  vectors:  individual  learner  and  unit  modeling,  instructional 
management  principles,  domain  modeling,  authoring  tools  and  methods, 
evaluation  tools  and  methods,  and  architectural  and  ontological  support  for 
adaptive  training.  The  reports  documenting  these  vectors  expand  the  scope  of  the 
adaptive  tutoring  research  described  in  ARL-SR-0284  (Sottilare  2013)  to  support 
ALM  requirements  in  the  mid-  and  long-term  evolution  of  training  and 
educational  technology:  the  Synthetic  Training  Environment  and  the  Future 
Holistic  Training  Environment  for  Live  and  Synthetic. 

This  report  (1  of  6  interdependent  research  outlines)  focuses  on  research  to  reduce 
the  time  and  skills  required  to  author  adaptive  training  and  education  content  in  a 
variety  of  militarily  relevant  domains.  Today,  most  intelligent  tutoring  systems 
(ITSs),  a  form  of  adaptive  training  tool  to  support  one-to-one  computer-based 
instruction,  support  well-defined  domains  in  mathematics  and  physics.  Since 
Soldiers  operate  in  more  complex,  dynamic,  and  ill-defined  domains,  it  is 
necessary  to  expand  the  scope  of  adaptive  training  tools  and  methods  to  support 
training  and  education  in  these  militarily  relevant  domains,  including  team 
training.  In  turn,  the  process  by  which  intelligent  tutors  are  currently  authored  for 
complex  domains  is  time  intensive,  requiring  a  degree  of  skill  similar  to  those  in 
computer  programming.  The  authoring  tools  and  methods  research  vector 
examines  the  various  tools,  processes,  and  technologies  that  enable  adaptive  tutor 
creation,  including  1)  software  interfaces  for  sequencing  content,  defining 
instructional  techniques,  strategies  and  tactics,  and  integrating  disparate  systems, 
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2)  the  creation  and  management  of  instructional  content  (i.e.,  content  management 
systems  and  reusable  learning  objects),  and  3)  algorithms  that  provide  for  semi-  or 
fully  automated  content  creation  with  significantly  reduced  human  input. 

2.  Research  Goals  and  Objectives 

The  goal  of  the  research  described  in  this  report  is  to  reduce  the  time  and  skill 
required  to  author  adaptive  instruction.  The  authoring  tools  and  methods  research 
vector  is  concerned  with  the  creation  and  management  of  various  resources 
required  to  support  adaptive  instruction  within  an  ITS  or  a  series  of  ITSs.  This 
report  outlines  research  objectives  intended  to  provide  guidelines,  methods,  tools, 
and/or  technologies  in  service  of  the  stated  research  goals.  The  primary  objectives 
of  this  research  are  intended  to  provide  guidelines,  best  practices,  tools,  models, 
and  methods  in  support  of  this  research  goal: 

•  For  human  authoring:  Apply  usability  heuristics  to  develop  best  practices 
to  guide  authoring  of  adaptive  systems  based  on  the  author’s  role  (e.g., 
domain  expert,  instructional  designer,  or  course  manager). 

•  For  computer-aided  authoring:  Identify  elements  of  the  authoring  process 
that  are  candidates  for  automated  or  semi-automated  authoring  processes 
to  remove  the  human  from  the  authoring  process  wherever  possible. 

•  For  leveraging  existing  training  and  educational  capabilities:  Establish 
standards  for  the  integration  of  functionally  disparate  tools  and 
technologies  (e.g.,  external  training  systems  and  serious  games)  that  may 
be  relevant  to  adaptive  systems  to  reduce  the  need  for  authoring. 

This  report  examines  the  background  and  requirements  for  authoring  adaptive 
training  and  educational  capabilities  in  different  domains  along  with  research 
challenges,  dimensions  of  authoring,  desired  end  states,  and  interdependencies 
with  other  adaptive  training  research  vectors. 

3.  Background 

While  human  tutoring  and  mentoring  are  common  teaching  tools,  current  US 
Anny  standards  for  training  and  education  are  group  instruction  and  classroom 
training — also  known  as  one-to-many  instruction.  Group  instruction  and 
classroom  training  have  been  generally  focused  on  acquiring  and  applying 
knowledge  in  proxies  for  live  training  environments  (e.g.,  desktop  simulations, 
virtual  simulations,  constructive  simulations,  and  serious  games). 
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Classroom  training,  especially  for  complex  topics,  is  often  taught  as  a  series  of 
lists  that  the  instructor  goes  through  in  a  linear  fashion  (Schneider  et  al.  2013). 
This  approach  puts  a  heavy  burden  on  the  learner  to  build  mental  models  and 
make  conceptual  connections.  Using  this  instructional  methodology  may  lead  to 
varying  degrees  of  success  due  to  individual  differences  in  skills,  traits,  and/or 
preferences.  Alternatively,  adaptive  training  can  support  those  individual 
differences  in  learners;  however,  the  burden  to  adapt  and  tailor  instruction  shifts 
to  the  author  (who  may  also  be  a  course  manager,  course  developer,  domain 
expert,  or  instructor)  to  build  mental  models  of  adaptive  training  components,  the 
relationships  between  the  components,  and  the  processes  (e.g.,  workflow)  by 
which  robust  tutors  are  created.  For  nonautomated  authoring  tasks,  the  creation 
and  use  of  these  mental  models  are  among  the  primary  research  challenges. 

Small  group  instruction  in  live  environments  has  also  been  used  to  assess 
application  of  knowledge  and  the  development  of  skills.  A  standard  feedback 
mechanism  for  US  Army  training  is  the  after-action  review  (AAR)  where 
significant  decision  points  and  actions  are  captured  for  small  group  discussion 
that  is  conducted  after  the  completion  of  a  training  event  to  help  capture  teachable 
moments  and  to  aid  Soldiers  in  reflecting  on  their  recent  training  experiences. 

Both  classroom  training  and  small  group  instruction  are  manpower  intensive, 
requiring  teachers,  mentors,  and  support  staff  to  guide  the  Soldier’s  experience. 
Today,  ITSs  primarily  guide  learner  training  and  education  for  cognitive  tasks  in 
well-defined  domains  (e.g.,  problem  solving  and  decision-making  tasks  in 
mathematics  and  physics).  Soldiers  tend  to  perform  cognitive,  affective, 
psychomotor,  and  social  tasks  in  both  well-defined  (e.g.,  building  clearing)  and 
ill-defined  domains  (e.g.,  leadership,  resource  allocation).  ITSs  generally  provide 
static  training  (e.g.,  sitting  at  a  desktop  computer  to  train  on  a  serious  game)  that 
falls  short  in  matching  the  dynamic  nature  of  many  US  Army  operational  tasks 
(e.g.,  psychomotor  tasks),  thereby  reducing  opportunities  to  develop  and  transfer 
skills  to  the  operational  environment. 

Research  is  needed  to  understand  the  characteristics,  similarities,  and  differences 
of  US  Anny  training  domains  (i.e.,  cognitive,  affective,  psychomotor,  social,  and 
hybrid)  to  develop  efficient  and  effective  adaptive  training  and  educational  tools 
and  methods  that  support  SRL  in  complex,  ill-defined,  and  physically  dynamic 
military  domains. 

3.1  Self-Regulated  Learning  and  the  US  Army  Learning  Model 

In  2011,  the  US  Army  placed  significant  emphasis  on  the  development  of  SRL 
skills  with  the  expectation  that  new  methods  of  instruction  (e.g.,  ITSs)  would 
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augment  institutional  training  (i.e.,  classroom  and  small  group  instruction).  One- 
to-one  human  tutoring  has  been  shown  to  be  significantly  more  effective  than 
one-to-many  instructional  methods  (e.g.,  traditional  classroom  instruction:  Bloom 
1984;  VanLehn  2011).  However,  it  is  neither  practical  nor  affordable  to  have  one 
expert  human  tutor  to  mentor  each  Soldier  in  the  US  Anny  for  every  required 
operational  task.  This  alone  signals  the  need  for  capabilities  to  support  one-to-one 
tailored  training  and  educational  experiences. 

Additionally,  under  the  ALM,  Soldiers  are  largely  responsible  for  managing  their 
own  learning,  but  SRL  skills  are  difficult  to  train  and  develop  (Butler  and  Winne 
1995;  Azevedo  et  al.  2009;  Graesser  and  McNamara  2010).  We  anticipate 
adaptive  training  tools  and  methods  will  fill  this  gap  and  will  provide  personalized 
guidance  to  acquire,  apply,  retain,  and  transfer  knowledge  and  skills  to  the 
operational  environment.  This  signals  the  need  for  a  computer-regulated  learning 
strategy  to  augment  missing  SRL  skills;  however,  adaptive  training  technologies 
must  first  become  affordable,  sufficiently  adaptive,  and  easy  to  use  for  this 
strategy  to  be  realized. 

3.2  Motivation  for  Research 

A  promising  alternative  to  one-to-one  human  tutoring  is  one-to-one  adaptive 
training  tools  that  include  ITSs.  Meta-analyses  and  reviews  support  the  claim  that 
ITS  technologies  routinely  improve  learning  over  classroom  teaching,  reading 
texts,  and/or  other  traditional  learning  methods.  These  meta-analyses  normally 
report  effect  sizes  (sigma  [a]),  which  refers  to  the  difference  between  the  ITS 
condition  and  a  control  condition  in  standard  deviation  units.  The  reported  meta¬ 
analyses  show  positive  effect  sizes  that  vary  from  a  =  0.05  (Dynarsky  et  al.  2007) 
to  a  =  1.08  (Dodds  and  Fletcher  2004),  but  most  hover  between  a  =  0.40  and  a 
=  0.80  (Fletcher  2003;  VanLehn  2011;  Graesser  et  al.  2012;  Steenbergen-Hu  and 
Cooper  2013,  2014;  Ma  et  al.  in  press).  Our  current  best  meta-meta  estimate  from 
all  of  these  meta-analyses  is  a  =  0.60.  This  performance  is  comparable  to  human 
tutoring,  which  varies  from  between  a  =  0.20  and  a  =  1.00  (Cohen  et  al.  1982; 
Graesser  et  al.  2011),  depending  on  the  expertise  of  the  tutor.  Human  tutors  have 
not  varied  greatly  from  ITSs  in  direct  comparisons  between  ITS  and  trained 
human  tutors  (VanLehn  et  al.  2007;  VanLehn  2011;  Olney  et  al.  2012). 

Graesser  et  al.  (2015,  in  press)  are  convinced  that  some  subject  matters  will  show 
higher  effect  sizes  than  others  when  comparing  any  intervention  (e.g.,  computer 
trainers,  human  tutors,  group  learning)  to  a  control.  It  is  difficult  to  obtain  high- 
effect  sizes  for  literacy  and  numeracy  because  these  skills  are  ubiquitous  in 
everyday  life  and  habits  are  automatized.  For  example,  Ritter  et  al.  (2007) 
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reported  that  the  Cognitive  Tutor  for  mathematics  has  shown  an  effect  size  of  a  = 
0.30-0.40  in  environments  with  minimal  control  over  instructors.  Human 
interventions  to  improve  basic  reading  skills  typically  report  an  effect  size  of  a  = 
0.20.  In  contrast,  when  the  student  starts  essentially  from  ground  zero,  such  as 
many  subject  matters  in  science  and  technology,  then  effect  sizes  are  expected  to 
be  more  robust.  ITSs  show  effect  sizes  of  a  =  0.60-2.00  in  the  subject  areas  of 
physics  (VanLehn  et  al.  2005;  VanLehn  2011),  computer  literacy  (Graesser  et  al. 
2004;  Graesser  et  al.  2012),  biology  (Olney  et  al.  2012),  and  scientific  reasoning 
(Millis  et  al.  2011;  Halpern  et  al.  2012).  As  a  notable  example,  the  Digital  Tutor 
(Fletcher  and  Morrison  2012)  improves  infonnation  technology  by  an  effect  size 
as  high  as  a  =  3.70  for  knowledge  and  a  =  1.10  for  skills.  The  effect  size 
attributed  to  improved  instruction  and  improved  domain  knowledge  has  not  been 
separated  in  this  analysis.  Such  large  effect  sizes  would  never  be  expected  in  basic 
literacy  and  numeracy. 

Overall,  these  are  promising  results  and  equate  to  an  increase  of  about  a  letter 
grade  improvement  over  traditional  classroom  instruction.  While  ITSs  are  a 
promising  technology  to  support  adaptive  training  for  individuals  in  well-defined 
domains  like  mathematics,  physics,  and  computer  programming,  the  US  Anny 
requires  the  ability  to  develop  and  exercise  Soldier  skills  in  more  ill-defined 
domains  (e.g.,  leadership)  and  at  the  unit  level  (e.g.,  collaborative  learning  and 
team  training).  Developing  and  maintaining  the  ability  to  make  effective  decisions 
under  stress  and  in  complex  environments  is  also  desirable. 

Adaptive  systems  by  their  nature  require  additional  content  and  complexity  to 
support  tailored  learning  for  each  user  and,  as  a  consequence,  have  a  very  high 
development  cost,  a  major  barrier  to  adoption  by  the  US  Anny.  Adaptive  systems 
are  also  insufficiently  adaptive  to  support  tailored  self-regulated  training  and 
educational  experiences  across  a  broad  spectrum  of  military  tasks  as  required  by 
the  ALM.  Today,  few  ITS  authoring  tools  are  generalized  across  all  of  the 
domains  requiring  training,  and  no  evaluation  criteria  or  standards  have  been 
developed  to  promote  reuse  and  interoperability  among  ITSs  (Sottilare  et  al. 
2012b).  In  other  words,  current  adaptive  systems  are  not  yet  intelligent  enough  to 
support  the  tailored  instruction  required  by  the  US  Army  in  the  breadth  of 
domains  being  trained;  but  there  is  a  stable  foundation  of  50  years  of  science  on 
which  to  grow  an  adaptive  training  and  education  capability  for  the  US  Army. 

3.3  Adaptive  Training  and  Education  Definitions 

In  support  of  the  ALM  and  affordable  adaptive  training  and  educational 
capabilities  for  the  US  Army,  ARL  is  investigating  and  developing  adaptive  tools 
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and  methods.  A  desired  end  state  is  the  automation  of  authoring  (creation) 
processes,  instruction,  and  evaluation  of  computer-regulated  training  and 
education  capabilities  to  help  build  SRL  skills  and  support  mixed-initiative 
interaction.  A  major  goal  within  this  research  program  is  to  reduce  the  time/cost 
and  knowledge/skill  required  to  author,  deliver,  and  evaluate  adaptive 
technologies  to  make  them  usable  by  a  larger  segment  of  the  US  Anny  training 
and  educational  community. 

Adaptive  training  and  education  research  includes  elements  of  adaptive  tutoring, 
distributed  learning,  virtual  humans,  and  training  effectiveness  evaluation.  For 
additional  detail  on  research  specific  to  ITSs,  refer  to  ARL-SR-0284  (Sottilare 
2013).  Definitions  are  provided  for  this  section  to  distinguish  between  adaptive 
training  and  education  elements  and  also  to  highlight  their  relationships: 

•  Adaptive  Tutoring:  Also  known  as  intelligent  tutoring;  tailored 
instructional  methods  to  provide  one-to-one  and  one-to-many  computer- 
guided  experiences  focused  on  optimizing  learning,  comprehension, 
performance,  retention,  reasoning,  and  transfer  of  knowledge  and  acquired 
skills  to  the  operational  environment. 

•  Adaptive  Tutoring  Systems:  Also  known  as  ITSs;  the  mechanism  or 
technologies  (tools  and  methods)  to  provide  tailored  training  and 
educational  experiences.  Adaptive  tutoring  systems  respond  to  changing 
states  in  the  learner  and  changing  conditions  in  the  training  environment  to 
optimize  learning.  Adaptive  tutoring  systems  anticipate  and  recognize 
teachable  moments. 

•  Virtual  Humans:  Artificially  intelligent  visual  representations  of  people 
that  simulate  or  emulate  cognitive,  affective,  physical,  and  social 
processes. 

•  Distributed  Learning:  Concurrent  distribution  of  training  and 
educational  content  to  multiple  users  at  the  point  of  need  in  which  content 
is  intelligently  selected  to  support  learning,  increased  performance,  and 
long-term  competency  in  selected  domains. 

•  Training/Learning  Effectiveness:  Evaluation  of  the  impact  of  training 
and  educational  tools  and  methods  on  usability,  learning,  comprehension, 
performance,  retention,  reasoning,  and  transfer  of  knowledge  and  acquired 
skills  to  the  operational  environment. 

•  Adaptive  Training  and  Education  Systems:  A  convergence  of  ITSs  and 
external  training  and  education  capabilities  (e.g.,  serious  games,  virtual 
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humans,  simulations)  to  support  engaging  experiences  with  reduced  need 
for  authoring  (Sottilare  2015). 

•  Generalized  Intelligent  Framework  for  Tutoring  (GIFT)  (Sottilare  et 
al.  2012a;  Sottilare  et  al.  2013):  An  open-source,  modular  architecture 
whose  goals  are  to  reduce  the  cost  and  skill  required  for  authoring 
adaptive  training  and  educational  systems,  to  automate  instructional 
delivery  and  management,  and  to  develop  and  standardize  tools  for  the 
evaluation  of  adaptive  training  and  educational  technologies. 

Adaptive  training  and  education  research  at  ARL  is  being  conducted  across  6 
interdependent  research  vectors:  individual  learner  and  unit  modeling; 
instructional  management  principles;  domain  modeling,  authoring  tools  and 
methods;  evaluation  tools  and  methods;  and  architectural  and  ontological  support. 
This  report  (1  of  6  interdependent  research  outlines)  focuses  on  authoring  tools 
research  for  adaptive  training  systems  with  the  goal  of  guiding  learning  in 
militarily  relevant  training  and  educational  domains. 

Soldiers  operate  in  a  variety  of  complex,  dynamic,  ill-defined  domains  where 
their  ability  to  persevere  in  the  face  of  adversity,  adapt  to  their  situation, 
collaborate,  and  think  critically  are  key  to  the  successful  completion  of  their 
assigned  missions.  To  develop  and  exercise  these  skills,  it  is  paramount  for 
Soldiers  to  train  in  challenging  environments.  Presently,  these  few  challenging 
training  environments  have  been  largely  provided  through  manpower-intensive 
methods  or  systems  with  little  ability  to  adapt  instruction  to  support  their  learning 
needs.  To  illustrate  this  point,  Franke  (2011)  asserts  that  through  the  use  of  case 
study  examples,  instruction  can  provide  the  pedagogical  foundation  for  decision 
making  under  uncertainty.  However,  this  approach  is  limited  in  implementation 
by  the  expanse  of  potential  cases  that  would  need  to  be  consistently  updated  and 
maintained  to  support  large  populations  like  the  US  Army. 

As  noted  previously,  adaptive  systems  like  ITSs  have  been  shown  to  be  effective 
in  promoting  learning  in  primarily  static  (e.g.,  learners  seated  at  desktop 
computers)  instructional  settings  within  relatively  simple,  well-defined  domains 
(e.g.,  mathematics,  physics)  for  individual  learners.  For  our  purposes,  static 
instruction  includes  cognitive,  affective,  or  social  training  tasks  where  a  desktop 
computer  delivers  instruction  and  where  the  physical  movement  of  the  learner  is 
limited  to  activities  that  can  be  conducted  while  seated.  For  example,  static 
instruction  can  effectively  support  cognitive  tasks  involving  decision  making  and 
problem  solving  but  are  less  effective  for  training  tasks  involving  motion  and 
perception  (e.g.,  land  navigation  and  marksmanship).  Ideally,  we  desire  portable 
adaptive  instructional  capabilities  to  go  with  Soldiers  to  support  training  and 
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education  at  their  point  of  need  across  a  wide  spectrum  of  US  Anny  operational 
tasks.  Research  is  needed  to  develop  tools  and  methods  to  support  broader  domain 
modeling,  which  is  representative  of  the  full  spectrum  of  US  Anny  operational 
tasks.  Standards,  interoperability,  and  automation  (e.g.,  automated  scenario 
generation)  (Zook  et  al.  2012)  will  likely  play  a  significant  role  in  making 
adaptive  training  practical.  In  this  way  adaptive  training  technologies  will  have 
the  greatest  impact  on  organizational  learning  in  the  US  Army. 

3.4  State  of  the  Art  in  Authoring  Tools  and  Methods 

While  positive  and  significant  gains  have  been  made  to  adaptive  tutor  authoring 
tools  and  technologies,  progress  is  often  measured  in  point-solutions  and  limited 
in  cross-domain  applicability.  A  number  of  significant  challenges  exist  within 
tutor  authoring. 

First,  given  the  current  state  of  the  art,  it  is  estimated  that  1  h  of  training  requires 
between  10  and  100+  h  of  development,  at  an  estimated  cost  of  $10,000  per  hour 
of  training  (Sottilare  and  Brawner  2014).  Where  authoring  systems  are  concerned, 
tradeoffs  are  often  made  between  functionality  and  generalizability.  For  instance, 
adaptive  tutoring  content  can  be  authored  for  well-structured  domains  (e.g.,  math, 
physics)  using  relatively  stable  authoring  tools  specifically  tailored  to  said 
content,  but  these  tools  fall  short  in  ill-defined  domains,  including  those  complex 
domains  that  are  relevant  to  Army  training  objectives  (e.g.,  decision  making, 
leadership,  teamwork).  It  logically  follows  that  authoring  content  for  Army¬ 
relevant  skills  and  tasks  would  currently  require  even  more  time  and  money  to 
author,  given  the  current  state  of  the  art.  Therefore,  as  advances  are  made  in  the 
parallel  areas  of  domain  modeling,  learner  modeling,  and  instructional 
management,  so  too  are  tools  needed  to  leverage  outputs  from  those  areas  to 
author  content  for  ill-defined,  complex  domains. 

Second,  the  domain  of  adaptive  tutor  authoring  is  still  in  its  infancy;  as  such,  there 
is  a  general  lack  of  standardization  between  tools  and  methods  (workflow)  for 
authoring  such  content.  While  ITS  and  traditional  computer-based  training  (CBT) 
bear  some  superficial  similarities  in  delivery  and  presentation,  the  complexity  of 
authoring  adaptive  tutoring  is  many  orders  of  magnitude  greater  because  of  the 
inclusion  of  learner  modeling,  pedagogical  agents,  complex  configuration,  and 
content  creation.  Further,  the  skill  of  authoring  adaptive  tutors  represents  a 
relatively  new  interaction  paradigm.  The  authoring  process  might  share  processes 
associated  with  other  content  creation  activities,  such  as  assembling  a  slide  deck, 
designing  a  blog  post,  or  sequencing  an  interactive  narrative  (such  as  a  video 
game).  Though,  none  of  the  aforementioned  activities  represent  a  one-to-one 
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mapping  with  authoring  adaptive  tutors.  It  is  a  new  form  of  content  creation  and 
an  activity  that  is  yet  to  be  fully  defined.  Thus,  potential  authors  must  form  new 
mental  models  regarding  the  creation  of  adaptive  tutors  by  integrating  new 
knowledge  with  existing  mental  models  of  activities  that  are  perceived  to  be 
similar.  The  authoring  vector  will  support  users  in  constructing  rich  and  accurate 
mental  models  of  tutor  authoring. 

Third,  the  manner  in  which  adaptive  tutors  will  interoperate  with  external  systems 
is  currently  piecemeal,  implemented  on  a  case-by-case  basis,  which  further 
complicates  the  authoring  process  and  serves  as  a  threat  to  the  generalizability  of 
adaptive  authoring  systems.  GIFT,  for  instance,  is  a  framework;  it  is  intended  to 
be  interoperable  with  a  variety  of  external  programs  and  technologies.  In  practice, 
separate  communication  pipelines  are  required  for  each  specific  application. 

Further,  these  pipelines  provide  inconsistent  functionality  with  respect  to  the  type 
of  data  that  can  be  exchanged  between  GIFT  and  the  application;  thus,  the 
communication  and  configuration  options  that  are  available  to  tutor  creators  will 
also  vary  wildly.  Interoperability  with  existing  simulators’  authoring 
environments,  content  management,  and  assessment  engines  is  a  significant  threat 
to  a  consistent  authoring  experience,  the  root  cause  of  which  may  lie  beyond  the 
scope  of  this  vector.  For  instance,  the  modeling,  simulation,  and  training  industry 
is  far  larger  in  size  and  scope  than  the  ITS  industry.  As  a  by-product,  efforts  to 
synchronize  the  modeling  and  simulation  community  with  the  needs  of  the  ITS 
community  have  languished  (Stottler  et  al.  2005)  because,  in  part,  developmental 
order  often  favors  a  simulation  first  and  instructional  system  second  approach. 
Those  authoring  concerns  are  not  limited  to  training  applications.  A  fully  realized, 
generalizable  authoring  system  will  also  need  to  be  able  to  interface  with  content 
management  systems,  learner  databases,  social  media  frameworks,  and 
(eventually)  intelligent  agents.  As  those  capabilities  are  enabled  by  the 
architecture,  research  will  be  required  to  define  authoring  requirements  in 
interacting  with  those  supporting  components. 

3.5  Overview  of  GIFT  Authoring  Tools 

GIFT  is  described  as  “an  empirically-based,  service-oriented  framework  of  tools, 
methods  and  standards  to  make  it  easier  to  author  computer-based  tutoring 
systems  (CBTS),  manage  instruction  and  assess  the  effect  of  CBTS,  components 
and  methodologies”  (Sottilare  et  al.  2012a).  As  GIFT  is  simultaneously  a  research 
project  and  open-source  application,  it  is  in  continuous  development  and  includes 
a  number  of  technologies,  features,  and  tools  targeted  toward  a  variety  of  users, 
including  instructional  designers,  researchers,  and  students.  For  instance,  the 
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framework  incorporates  models  of  domain  content,  pedagogical  methods,  sensor 
data  processing,  interoperability  with  external  applications,  and  learners,  each 
with  its  own  module  and/or  configuration.  As  such,  it  is  important  to  recognize 
that  GIFT  is  not  simply  a  set  of  authoring  tools;  however,  research  issues  related 
to  authoring  are  part  of  the  focus  of  the  current  program  plan  and  the  explicit 
focus  of  this  report. 

Until  recently,  GIFT  applications  were  created  by  writing  and/or  editing 
extensible  Markup  Language  (XML).  One  “tutor”  was,  in  reality,  a  series  of 
separately  configured  but  inter-reliant  components.  Directly  editing  the  files  was  a 
complex  process  that  required  users  to  understand  how  to  properly  write  encoded 
XML.  XML  editing  programs  were  created  to  make  its  editing  easier,  but  this  did 
not  alleviate  users’  need  to  understand  how  the  various  components  of  GIFT 
interact  with  one  another  at  the  system  level.  By  comparison,  the  current  stable 
versions  of  GIFT  (2014-2  and  2015-1)  provide  some  browser-based  interfaces  to 
facilitate  semi-automated  user  creation  of  the  XML  output;  these  interfaces  take 
the  first  steps  toward  unifying  a  diverse  set  of  authoring  tools  behind  a  consistent 
user  interaction  experience.  However,  the  current  implementation  of  the 
graphical-user  interfaces  does  not  change  the  requirement  that  authors  must 
understand  the  system-level  conceptual  model  of  GIFT.  In  this  report  we  propose 
research  that  will  result  in  interfaces  that  help  to  close  the  gap  between  an 
author’s  mental  model  of  adaptive  tutor  creation  and  the  system-level  conceptual 
model  of  GIFT. 

Sottilare  (2013)  outlined  goals  for  GIFT,  including  the  following  authoring- 
related  goals:  decrease  the  effort  required  for  authoring  (time,  cost,  etc.),  decrease 
the  skill  threshold  required  to  author  adaptive  tutoring,  support  users  in  organizing 
knowledge,  support  pedagogical  design,  allow  for  rapid  prototyping  of  adaptive 
tutoring,  leverage  standards  for  integration  of  external  media  and  applications, 
and  promote  content  reuse.  The  authoring  goals  for  GIFT  are  concurrent  with 
authoring  goals  for  ITS,  in  general,  which  endeavor  to  support  training  and 
education  requirements  outlined  in  the  ALM. 

4.  US  Army  Requirements  for  Adaptive  Training  Systems  and 
Authoring 

The  Anny  Science  and  Technology  community  uses  Warfighter  Outcomes 
(WFOs)  as  the  authoritative  source  for  identifying  Warfighter  needs.  WFOs  are 
used  to  share  research  and  future  technology  solutions.  In  the  training  and 
education  domain,  the  adaptive  training  and  education  research  program  is 
targeting  4  specific  requirements  to  support  the  evolution  of  US  Army  training: 
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adaptive  training  and  education  systems;  big  data;  training  at  the  point  of  need; 
and  artificial  intelligence. 

4.1  Adaptive  Training  and  Education  Systems  and  Authoring 

The  primary  gap  to  be  addressed  under  this  Army  requirement  is  the  lack  of 
adaptive  systems  (e.g.,  intelligent  tutors)  to  support  individual  and  collective 
(team  or  unit)  training.  The  Anny  needs  an  adaptive  training  and  education 
capability  that  is  persistent  and  easy  to  use/access  with  minimal  start-up  time. 
There  are  also  requirements  to  automate  an  informal  AAR  (also  known  as  post¬ 
exercise  critique)  to  reduce  the  time  and  skills  needed  to  produce  the  AAR  and 
improve  its  focus  and  quality.  Another  line  of  thought  also  notes  that  the  artificial 
intelligence  in  ITSs  could  be  used  to  facilitate  rapid  mission  planning  and  course- 
of-action  analyses  as  a  job  aid  in  operational  contexts. 

The  major  connection  between  the  adaptive  training  and  education  requirement 
and  the  authoring  tools  and  methods  research  vector  is  the  need  to  extend 
authoring  capabilities  in  2  directions.  The  first  of  these  directions  relies  upon  the 
enabling  of  the  authoring  of  feedback  for  AAR  in  complex  domains  or  in  the 
rapid  authoring  of  correct  assessment  models  within  simplistic  domains.  The 
second  of  these  directions  relies  upon  the  automation  and  magnification  of  the 
authoring  process  through  extensive  reuse,  content  readability  by  computerized 
process,  and  the  configuration  of  the  system  to  meet  the  needs  of  the  learner. 

4.2  Big  Data  and  Authoring 

The  primary  gap  to  be  addressed  under  this  US  Army  requirement  is  that  there  is 
a  lack  of  capability  to  handle  and  process  large  amounts  of  structured  and 
unstructured  data  (also  referred  to  as  big  data).  One  capability  needed  is  a 
structured  data  analytics  program  linking  individual  data  (e.g.,  achievements)  to 
required  long-term  competencies  in  military  occupational  specialties  (MOSs). 
This  would  allow  Soldiers  to  understand  where  they  rank  in  terms  of  experiences 
and  achievements  among  other  Soldiers  in  their  MOS.  It  would  also  allow  the  US 
Anny  to  identify  specific  experiences  among  successful  Soldiers  in  that  MOS  and 
provide  a  model  for  other  Soldiers  in  that  MOS  to  follow.  The  data  could  also  be 
used  by  course  managers  and  instructors  to  continuously  improve  instruction  and 
the  mental  models  of  both  human  and  computer-based  instructors.  Finally,  data 
collected  on  trainee  learning  and  performance  during  adaptive  training 
experiences  could  be  used  to  facilitate  Unit  Training  Management  where  unit 
commanders  would  have  access  to  empirical  data  to  support  unit  training 
decisions. 
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The  major  connection  between  the  Anny’s  big  data  requirement  and  authoring 
tools  is  ability  to  collect  or  generate  learner  data  from  content  interactions, 
content  data  for  machine  traversal,  and  training  environment  data  to  make  the 
connection  among  learner  behaviors,  instructional  tactics,  and  their  varying 
degrees  of  effectiveness  given  instructional  context  (existing  conditions).  This 
will  allow  course  managers  to  be  aided  by  machine  techniques  of  content  creation 
and  to  see  the  effectiveness  of  their  authored  material. 

4.3  Training  at  the  Point  of  Need  and  Authoring 

The  primary  gap  to  be  addressed  under  this  US  Anny  requirement  is  the  lack  of 
an  easily  accessible,  persistent,  cost-effective,  and  low-overhead  training 
environment.  A  capability  is  needed  to  bring  training  to  Soldiers  instead  of 
Soldiers  going  to  fixed  training  locations.  This  point-of-need  training  capability 
would  be  easily  distributed,  web  based,  and  built  upon  an  open-enterprise 
architecture  in  the  cloud.  US  Anny  training  and  educational  opportunities  would 
be  available  on  demand  anywhere  and  anytime.  However,  the  delivery  mechanism 
(e.g.,  laptop  computer,  mobile  device,  and  smart  glasses)  for  adaptive  training  is 
critical  in  detennining  the  limitations  of  the  domain  model  scope  and  complexity. 
For  example,  it  may  be  extremely  difficult  to  train  all  the  complexities  of  a 
psychomotor  task  in  a  desktop  computer  setting. 

The  major  connection  between  point-of-need  training  and  authoring  relies  upon 
the  practicality  of  extending  adaptive  training  beyond  the  desktop.  This  is 
primarily  an  architectural  content  conversion  process,  avoiding  the  work  of 
making  multiple  complete  instances  of  content  (web,  mobile,  desktop,  smart 
glasses,  etc.).  Training  content  (e.g.,  instructional  content,  assessments,  feedback) 
should  be  able  to  be  reworked  into  the  appropriate  format. 

4.4  Artificial  Intelligence  (Al)  Capabilities  and  Authoring 

The  primary  gap  to  be  addressed  under  this  US  Anny  requirement  is  that  the  US 
Anny  lacks  an  automated  capability  to  replicate  the  complexity  and  uncertainty  of 
the  operational  environment.  This  gap  specifically  points  to  the  lack  of 
adaptiveness  in  virtual  humans,  intelligent  tutoring  systems,  and  other  training 
capabilities.  This  gap  leads  to  Soldiers  developing  training-response  strategies 
that  result  in  less  challenging  training  over  time  along  with  lower  engagement  and 
lower  levels  of  learning  and  transfer  of  skills  to  more  challenging  operational 
environments. 

The  major  connection  between  AI  capabilities  and  authoring  involves  the 
discovery  and  innovation  of  techniques  to  support  computer-aided  authoring. 
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Computer-aided  authoring  comes  in  a  few  differing  types,  from  automated 
scenario  evolution,  to  automated  question  and  hint  generation,  to  intelligent 
content  search  and  retrieval.  By  implementing  such  concepts  with  AI  support,  the 
authoring  burden  for  instructors  and  course  managers  developing  highly  complex 
training  and  educational  domains  may  be  reduced. 

AI  capabilities,  domain  modeling,  and  authoring  involve  the  discovery  and 
innovation  of  techniques  to  support  a  concept  called  “automated  scenario 
evolution”  being  developed  by  Sottilare  (2015).  AI  capabilities  are  needed  to 
support  automated  scenario  evolution  where  AI  drives  the  generation  of  new 
“child”  scenarios  from  a  single-parent  scenario  based  on  dimensions  of  that 
scenario  and  the  state  of  the  trainee.  In  this  way,  the  authoring  burden  for  highly 
complex  training  and  educational  domains  may  be  reduced. 

For  example,  consider  a  single  scenario  where  dimensions  include  variable 
challenge  levels  based  on  3  threats  (i.e.,  low,  moderate,  high),  3  types  of  field  of 
view  (i.e.,  narrow,  moderate,  and  wide),  and  clear  line  of  sight  (i.e.,  near, 
moderate,  and  far).  AI  could  spawn  27  new  child  scenarios  based  on  combinations 
of  these  variables.  This  requirement  is  closely  linked  to  adaptive  training 
capabilities  described  in  Section  4.1  of  this  report,  and  the  realization  of  this 
capability  will  enable  the  development  of  affordable  self-authoring  adaptive 
systems.  Through  this  capability,  complex  domains  may  be  modeled  for  adaptive 
training  systems  without  the  need  for  long  development  cycles  or  special 
authoring  skill  sets. 

5.  Understanding  the  Dimensions  of  Authoring 

There  are  4  typical  elements  that  compose  ITSs,  a  prime  example  of  an  adaptive 
training  and  education  system:  a  learner  or  trainee  model,  an  instructional  or 
pedagogical  model,  a  domain  model,  and  some  type  of  user  interface.  The  domain 
model  typically  includes  an  expert  or  ideal  student  model  by  which  the  adaptive 
system  measures/compares/contrasts  the  progress  of  the  learner  toward  learning 
objectives.  The  domain  model  also  includes  the  training  environment,  the  training 
task,  and  all  of  the  associated  instructional  actions  (e.g.,  feedback,  questions, 
hints,  pumps,  and  prompts)  that  could  possibly  be  delivered  by  the  adaptive 
system  for  that  particular  training  domain.  Typical  interaction  among  the  learner, 
the  training  environment,  and  the  adaptive  system  (tutoring  agents)  is  shown  in 
Fig.  1. 

Typical  training  systems  examine  the  interaction  between  the  learner  and  the 
training  environment  to  measure  progress  toward  learning  objectives.  The  learner 
acts  on  the  environment  (e.g.,  opens  a  door  or  makes  a  choice  to  move  into  the 
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room  or  stay  outside)  and  then  observes  any  changes  or  reactions  within  the 
environment.  Adaptive  systems  add  a  layer  of  software-based  tutoring  agents  that 
are  designed  to  guide  the  learner  in  much  the  same  way  as  a  human  tutor  interacts 
with  a  learner.  The  tutoring  agents  observe  the  behaviors  of  the  learner  to  assess 
their  states  (e.g.,  performance  and  attitudes)  and  interact  with  the  learner  to 
provide  support,  direction,  and  instruction.  In  addition,  they  track  the  effect  of 
interactions  on  learning.  Tutoring  agents  also  interact  with  the  training 
environment  and  may  manipulate  the  environment  to  present  more  challenging  or 
less  challenging  scenarios  in  response  to  the  assessed  state  of  the  learner. 


Tutoring  Agents 


agent 

observes 

environment 


! 

A 

agent 

agent 

agent  acts  observes  & 

interacts 

to  change  assesses 

with 

environment  learner 

learner 

> 

t 

t 

agent 
observes 
effect  on 
learning 


Training 

Environment 


learner  acts  on  environment 

< - 

learner  observes  environment 

- > 


Learner 


Fig.  1  Adaptive  training  interaction 

Authoring  is  an  element  of  the  ITS,  which  is  not  overtly  visible  in  Fig.  1,  but  it 
potentially  interacts  with  and/or  influences  each  of  the  other  elements  within  the 
tutor.  Authoring  identifies,  for  example,  what  is  observed,  how  an  agent  interacts, 
and  why  it  selects  certain  interaction  strategies  over  others.  In  practice,  the 
authoring  element  requires  input  from  many  dimensions:  the  ability  to  configure 
each  of  the  other  ITS  elements  (including  external  training  environments),  a 
mechanism  to  incorporate  new  components  and  configurations  into  the  ITS,  and 
significant  human  capital  (at  least  for  the  near  future)  to  help  shape  the  tutor, 
manipulate  its  many  configurations,  and  harness  its  output.  These  dimensions  are 
described  in  greater  detail  below. 

The  widespread  adoption  of  adaptive  tutoring  will  require  authoring  systems  that 
are  easy  to  use  and  contain  the  functionality  needed  to  support  a  wide  variety  of 
training  tasks  and  environments.  Murray  (2003)  described  the  current  state  of 
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adaptive  tutor  authoring  as  a  series  of  tradeoffs  between  usability,  depth,  and 
flexibility.  Given  the  depth  and  flexibility  required  to  author  for  the  complex 
domains  associated  with  Anny  training  objectives,  the  current  authoring  process 
is  complicated,  requiring  a  degree  of  skill  similar  to  that  of  computer 
programming. 

Mental  models  will  serve  as  the  underlying  theory  for  the  user-centered  research 
component  of  this  outline  (described  in  Section  6.1).  Rouse  and  Morris  (1986) 
explained  that  mental  models  “are  the  mechanisms  whereby  humans  are  able  to 
generate  descriptions  of  purpose  and  fonn,  explanations  of  system  functioning 
and  observed  system  states,  and  predictions  of  future  states”  (p.  7).  Mental  models 
influence  users’  expectations  regarding  a  system’s  functionality  and  guide  user 
interaction  behavior  (Ososky  2013).  An  individual’s  mental  model  regarding  a 
particular  system  is  influenced  by  past  experiences  and  perceived  similarity  of 
other  systems  to  the  target  system.  Further,  human  mental  models  do  not  need  to 
be  complete  or  even  accurate  to  be  applied  to  a  specific  system  interaction 
(Nonnan  1986). 

Therefore,  to  increase  the  usability  of  authoring  tools,  it  first  is  necessary  to 
identify  and  describe  authors’  mental  models  for  authoring  tools.  Potential 
authors,  however,  may  enter  into  an  interaction  with  authoring  tools  with  a  variety 
of  backgrounds,  including  instructional  design,  training  facilitation,  and  subject 
matter  expertise.  It  is  likely  that  each  user  group  will  reveal  a  mental  model  for 
interacting  with  authoring  tools,  which  is  significantly  different  from  the  others.  It 
is  important  to  acknowledge  the  distinction  between  user  groups  because  those 
models  are  likely  to  be  (even  more)  different  from  the  manner  in  which  the 
authoring  system  is  actually  designed.  This  is  because  engineers  have  their  own 
mental  models  about  how  authoring  tools  should  work,  and  they  use  this  model  to 
develop  the  realized  version  of  the  system  (Streitz  1988).  The  engineer’s  or 
designer’s  model  has  been  referred  to  as  the  conceptual  model  (Nonnan  1983). 

Stated  differently,  users  may  know  (in  their  minds)  what  type  of  adaptive  content 
they  need  to  author  but  are  unable  to  leverage  the  authoring  system  to  achieve 
their  goals.  Thus,  a  gap  exists  between  user  intent  and  system  usability.  Likewise, 
the  more  that  is  assumed  about  how  users  may  interact  with  authoring  tools,  the 
larger  this  gap  is  likely  to  be.  Nonnan  (1986)  described  this  gap  as  the  gulfs  of 
execution  and  evaluation.  The  gap  between  user  intent  and  system  representation 
was  expressed  as  2  gulfs  because  there  are  2  ways  to  close  the  gap.  Users  can 
adapt  their  plans  to  the  design  of  the  system  (e.g.,  acquire  new  knowledge  of  a 
complex  system),  or  the  system’s  interface  can  be  improved  to  match  the  needs  of 
the  user  (e.g.,  leverage  familiar  terms  and  interaction  elements).  For  adaptive  tutor 
authoring  tools,  it  will  likely  be  necessary  to  approach  this  gap  from  both  sides. 
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6.  Authoring  Research  Goals  and  Challenges 


A  foundational  goal  of  adaptive  training  and  education  research  at  ARL  is  to 
model  the  perception,  judgment,  and  behaviors  of  expert  human  tutors  to  support 
practical,  effective,  and  affordable  learning  experiences  guided  by  computer- 
based  agents.  To  this  end,  and  in  support  of  the  authoring  goals  stated  earlier,  the 
following  4  primary  challenges  for  authoring  adaptive  systems  have  been 
identified  and  are  described  in  this  section. 

•  Describe  human  mental  models  and  define  authoring  interaction 
paradigms. 

•  Identify  candidates  for  automated  authoring  and  develop  processes  to 
reduce  the  human  authoring  workload. 

•  Extend  authoring  capabilities  to  support  integration  with  existing  training 
and  educational  technologies  (e.g.,  training  simulations,  simulators,  and 
serious  game  platforms). 

•  Enable  collaborative  authoring. 

6.1  Describe  Human  Mental  Models  and  Define  Authoring 
Interaction  Paradigms 

Stated  previously,  authoring  adaptive  tutoring  content  represents  a  new  interaction 
paradigm.  Further,  tradeoffs  are  made  in  authoring  between  usability,  depth,  and 
flexibility.  The  purpose  of  this  subtask  is  to  attend  to  the  usability  component  of 
the  current  state  of  the  art.  The  end  goal  of  this  subtask  is  to  reduce  the  time  and 
skill  required  to  author  adaptive  tutoring  content. 

Designing  an  interface  to  support  the  creation  of  adaptive  tutors  is  a  nontrivial 
task  and  one  that  is  not  quite  like  many  other  superficially  similar  tasks  (e.g., 
creating  slides,  creating  a  web  page,  or  designing  a  video  game).  Many  potential 
users  may  not  even  fully  realize  the  benefits  that  adaptive  tutoring  can  provide 
over  traditional  classroom  or  CBT.  Further,  there  may  be  multiple  user  groups 
that  may  be  interested  in  creating  adaptive  tutors,  and  each  of  these  user  groups 
has  their  own  sets  of  knowledge  and  interaction  expectations.  For  instance,  we 
can  theoretically  identify  potential  authors  from  instructional  designers,  subject 
matter  experts,  and  researchers,  respectively,  but  do  they  need  to  be  regarded  as 
different  types  of  users  for  the  authoring  system? 

Therefore,  the  research  approach  will  examine  the  design  of  authoring  tools  from 
both  sides  of  the  system  usability  gap.  We  intend  to  conduct  user  research  to 
illustrate  the  various  mental  models  of  adaptive  tutor  design,  as  well  as  non- 
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system-specific  goals  of  those  potential  users.  We  will  leverage  users’  goals, 
desires,  and  intentions  to  drive  the  requirements  for  the  design  of  a  prototype 
authoring  tool  interface  for  GIFT.  Outputs  from  this  research  will  also  help  to 
identify  differences  in  the  needs,  if  any,  between  each  user  group.  Research 
activities  may  include,  but  are  not  limited  to,  interviews,  surveys,  and 
direct/indirect  observation.  Authoring  tools  will  be  evaluated  along  dimensions  of 
utility  (“Does  the  system  support  users’  goals?”),  usability  (“Can  the  user  easily 
accomplish  their  goals?”),  and  the  time  to  author  an  hour  of  adaptive  course 
content. 

It  is  unlikely,  at  least  in  the  near  term,  that  it  will  be  possible  to  design-out  all  of 
the  complexity  of  authoring  an  adaptive  tutor.  We  must  strive  for  a  reasonable 
balance  in  the  tradeoffs  between  usability,  depth,  and  flexibility.  To  that  end, 
users  may  need  to  learn  about  concepts  that  are  unique  to  ITS  in  order  to  develop 
new  skills  and  knowledge  that  will  enable  users  to  design  effective  tutors  (for  a 
discussion  of  automated  authoring,  see  Section  3.2).  To  that  end,  we  propose  to 
examine  the  system  side  of  the  usability  gap  as  well.  Currently,  GIFT  is 
composed  of  a  rudimentary  set  of  GUI-based  authoring  tools  defined  by  system- 
level  terminology  and  theoretical  ITS  components.  We  will  conduct  research  that 
will  examine  conceptual  models  of  tutor  authoring  with  the  goal  of  improving  the 
ease  of  learning  of  terminology  and  components  associated  with  ITS.  In  service  of 
this  objective,  we  will  survey  the  waterfront  of  existing  authoring  tools  and 
technologies  to  identify  themes  in  authoring  components  and  tenninology,  as  well 
as  present  suggestions  for  improvements  upon  the  current  workflow.  The  outputs 
from  this  research  are  expected  to  produce  guidelines  and  best  practices  that  could 
be  used  to  drive  the  establishments  of  standards  in  the  field  of  adaptive  tutoring. 

Continuous  improvement  of  prototype  authoring  interfaces  and  interaction 
paradigms  is  the  final  cornerstone  of  our  proposed  research  approach.  We  will 
track  the  progress  of  the  authoring  system  after  the  bridge  has  been  built  from 
both  sides  of  the  gap.  We  intend  to  develop  and  implement  tools  that  will 
manually  and  automatically  collect  data  with  respect  to  the  use  of  and  experience 
with  GIFT  authoring  tools.  Research  activities  may  include,  but  are  not  limited  to, 
website  metrics  (in  the  case  of  cloud-based  authoring  tools),  remote  unattended 
usability  testing,  and  collection  of  qualitative  user  experience  feedback  via  the 
authoring  system  or  GIFT  community  forum  channels. 


17 


6.2  Identify  Candidates  for  Automated  Authoring  and  Develop 
Processes  to  Reduce  the  Human  Authoring  Workload 

The  process  of  building  a  tutor  is  more  involved  than  the  creation  and 
customization  of  its  various  component  parts.  Many  of  these  component  parts  will 
not  require  authoring  for  the  average  user.  As  an  example,  the  pedagogical 
method  of  instruction  from  one  domain  of  instruction  to  another  should  remain 
mostly  the  same.  The  same  may  be  said  of  the  methods  of  modeling  the  learner, 
or  of  the  sensors  used.  Having  an  authoring  system  that  makes  extensive  use  of 
default  settings  (or  preconfigured  sets  of  settings)  allows  the  individual  tutor 
author  to  focus  on  the  domain  of  instruction. 

In  regards  to  the  domain  of  instruction,  the  basic  process  is  to  1)  gather  content, 
2)  make  the  content  available,  3)  customize  it  to  suit  the  individual  needs  of  the 
audience,  4)  generate  tutoring  information,  5)  perform  the  delivery  of  the  content 
for  informational  and  practice  settings,  and  6)  take  traditional  ITS  actions  (e.g., 
model  the  learner,  select  from  content,  track  learner  experience)  (Olney  et  al. 
2015). 

Currently,  the  first  step  of  authoring  (content  gathering)  is  typically  performed  as 
a  series  of  searches  via  a  search  engine.  Modern  search  engines  perfonn  many  of 
the  functions  behind  this  content  gathering  automatically  (utilizing  search  history, 
individual  profiles,  etc.)  in  an  effort  to  improve  search  results.  If  available  content 
were  already  indexed  according  to  keywords  and  learning  objectives,  it  is 
conceivable  that  this  process  may  be  automated  (Jesukiewicz  and  Rehak  2011; 
GooruLeaming  2014). 

The  actions  of  indexing  this  content  for  availability,  such  as  tagging  the  content 
with  metadata,  can  be  perfonned  in  an  automated  fashion  with  today’s  cutting- 
edge  technology  (Veden  2015),  which  is  an  architecturally  assisted  process 
discussed  in  another  research  vector.  If  this  content  is  text  related,  simple  tutoring 
infonnation,  such  as  hint/prompt/pump  generation,  can  be  performed  in  a 
machine-assisted  manner  (Olney  et  al.  2012). 

Indexed  and  actionable  content,  hosted  in  a  repository,  can  be  traversed  through  a 
machine-assisted  process  in  accordance  with  the  learner  model.  As  an  example, 
content  for  a  highly  motivated  and  experienced  learner  could  be  queried  from  a 
content  provisioning  system  with  descriptors  of  the  context  needed  for  the 
individual  (Ray  et  al.  2014).  Such  a  provisioning  system  would  be  able  to  match  the 
learner  to  content  based  on  his/her  personal  traits  and  the  individual  content 
attributes  (e.g.,  subject  matter,  difficulty  level,  user  ratings).  A  learner  can  progress 
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through  a  learning  path  planned  by  the  pedagogical  system,  taking  prescribed 
pedagogical  actions,  such  as  hint  giving,  because  of  their  pregenerated  nature. 

The  system  above  describes  a  process  of  automating  the  process  of  authoring  and 
automatically  building  an  appropriate  course,  using  up-to-date  and  reusable 
learning  objects  (RLOs)  created  and  curated  in  a  centralized  repository.  Portions 
of  such  a  system  exist  through  pedagogical  recommendation  engines,  such  as  the 
eMAP  (Goldberg  et  al.  2012),  content  curation  systems,  such  as  Nuxeo  (Nuxeo 
Platform  2015),  exchange  languages,  such  as  xAPI  (Regan  2013),  and  3QL  (Ray 
et  al.  2014),  and  others.  However,  research  is  needed  to  support  the  tie-in, 
integration,  testing,  effectiveness,  and  accuracy  of  such  an  automated  system. 

6.3  Extend  Authoring  Capabilities  to  Support  Integration  with 
Existing  Training  and  Educational  Technologies _ 

The  development  of  an  ITS  is  expensive,  labor  intensive,  and  requires  highly 
skilled  personnel  from  a  variety  of  disciplines.  One  way  to  reduce  the  time  and 
cost  of  development  is  to  encourage  reuse  through  the  use  of  standards  for 
processes,  tools,  and  software  components.  Templates  for  domain  models  and 
learning  content  could  also  reduce  costs  (Sottilare  2014).  Content  Management 
Systems  (CMSs),  for  example,  can  be  used  to  reduce  costs  by  supporting  learning 
content  reuse.  However,  it  is  time  consuming  to  upload  content,  and  typically, 
metadata  is  manually  input  by  administrative  personnel,  reducing  the 
effectiveness  of  search  and  retrieval  tools.  Research  continues  in  automating  the 
process  of  analyzing  documents  to  decompose  them  into  RLOs  and  generate 
accurate  metadata.  Data  analytics  is  another  method  that  may  be  used  to  provide  a 
smart  search  capability  for  a  CMS.  Current  research  is  also  exploring  “best 
practices”  for  the  use  of  RLOs  to  include  the  use  of  assessments.  Social  media 
may  also  provide  a  crowd-sourced  capability  in  the  generation  of  learning  content, 
as  well  as  vetting  current  learning  content/courses. 

In  adaptive  training  environments,  virtual  humans  can  be  used  as  virtual 
instructors  (e.g.,  talking  heads)  to  help  guide  the  learner  through  a  training 
session.  Nonplayer  Characters  can  also  be  used  to  support  training  scenarios  that 
require  human  interaction.  GIFT  provides  the  Media  Semantic  character  set  to 
support  the  use  of  commercial  virtual  humans.  Further  research  is  required  to  ease 
the  authoring  process  for  the  creation  and  management  of  virtual  humans  within 
adaptive  tutoring  environments. 

Another  challenge  is  to  improve/standardize  authoring  support  for  external 
training  applications  (e.g.,  the  integration  of  ITSs  with  simulations  or  game -based 
training  environments).  The  Game -based  Architecture  for  Mentor-Enhanced 
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Training  Environments  (GAMETE)  is  one  example.  GAMETE  is  focused  on 
allowing  training  developers  to  add  an  intelligent  tutoring  capability  into  a  game- 
based  simulation  environment  (Engimann  et  al.  2014).  Automated  scenario 
generation  (Zook  et  al.  2012)  is  another  area  of  interest,  as  is  authoring  support 
for  tutors  intended  for  units,  teams,  and  teams  of  teams. 

In  addition  to  the  areas  discussed  above,  research  is  needed  to  determine  optimal 
methods  for  authoring: 

•  Techniques,  strategies,  and  tactics  within  GIFT 

•  Perfonnance  and  competency  assessments  within  GIFT 

•  Domain  content  and  expert  model  development 

•  New  content  and  instructional  models  from  existing  data  sources 

6.4  Enable  Collaborative  Authoring _ 

It  is  envisioned  that,  in  many  cases,  a  team  of  individuals  will  be  developing  a 
course;  those  individuals  will  have  differing  responsibilities.  As  an  example,  one 
individual  may  create  a  pre-/posttest  while  another  may  put  together  lectures, 
another  may  design  practice  scenarios,  and  yet  another  may  review/test  the 
completed  course  for  quality  assurance.  Beyond  the  user  experience  concerns  for 
individual  authoring  discussed  in  the  previous  section,  there  are  a  number  of 
follow-on  issues  that  will  need  to  be  addressed  to  enable  collaborative  authoring. 
As  a  simple  example,  content  permissions  need  to  be  implemented  that  allow  the 
appropriate  individuals  (or  groups)  to  have  access  to  different  components,  keep 
items  restricted  from  general  users,  prevent  overwriting  of  changes  from 
unauthorized  users,  and  provide  a  functional  baseline  for  quality  testing.  This 
research  subtask  will  examine  issues  related  to  collaborative  authoring,  including 
implementation,  compatibility,  and  exporting. 

6.4.1  Web  Authoring 

In  GIFT’S  desktop  implementation,  authors  can  create  their  own  questions, 
surveys,  and  survey  contexts  using  the  Survey  Authoring  System.  All  of  the 
questions  that  have  been  created  and  included  in  GIFT’S  baseline  are  available  to 
all  GIFT  users.  Once  an  individual  creates  his/her  own  surveys  and  questions,  that 
content  stays  local  to  the  user’s  own  desktop  version  unless  he/she  exports  and 
then  imports  the  survey  to  another  individual’s  version  of  GIFT.  In  the 
development  of  a  web-based  version  of  the  Survey  Authoring  System, 
consideration  will  need  to  be  given  to  where  content  created  by  an  individual  user 
is  stored  and  whether  it  is  included  in  the  survey  bank  for  all  to  use.  This  creates  a 
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unique  problem,  as  instructors  who  may  be  using  GIFT  to  create  and  administer 
tests  likely  will  not  want  their  assessments  and  questions  (and  answers)  readily 
available  to  other  individuals  and  students. 

To  address  the  storage  and  availability  of  content  authored  by  specific  individuals, 
research  and  thought  will  need  to  be  put  into  permissions  and  storage.  The  survey 
authoring  system  in  particular  will  likely  need  to  only  provide  the  stored  content 
to  the  individual  that  created  it  and  anyone  working  on  the  research/authoring 
team.  Consideration  will  need  to  be  given  to  the  new  challenges  that  develop  from 
moving  the  authoring  tools  to  a  web  environment.  Development  will  be  needed  to 
address  the  traditional  problems  of  sharing,  access,  and  editing. 

6.4.2  Versioning  and  Compatibility 

It  is  very  important  that  created  content  is  backward  compatible  with  previous 
versions  of  GIFT.  Individuals  may  invest  a  large  amount  of  time  in  developing  a 
course  in  a  specific  version  of  GIFT  and  may  resist  moving  to  a  newer  version  if 
their  content  will  not  be  compatible.  In  a  desktop  implementation  they  can 
continue  using  their  previous  version;  however,  in  a  web-based  implementation, 
this  backward  compatibility  has  greater  importance.  Web-based  implementations 
do  not  necessarily  allow  for  the  functionality  of  “snapshotting”  a  course  in  order 
to  maintain  an  existing  version. 

Furthermore,  there  is  an  importance  on  optional  updates  of  course  material.  As  an 
example,  if  a  resource  used  within  a  course  has  been  updated  by  a  third  party,  the 
course  author  should  have  the  capability  (but  not  the  necessity)  to  update  that 
course  to  use  the  new  object.  Such  updates  should  not  be  mandatory,  as  the  course 
author  should  have  the  opportunity  to  review  the  material  prior  to  delivering  it  to 
students  from  a  shared  web-enabled  environment. 

6.4.3  Sharing  and  Exporting  Tutors 

In  addition  to  collaborative  authoring,  it  is  logical  to  assume  that  authors  will 
want  to  share  their  created  tutors  with  schoolhouses  and/or  learners.  GIFT 
currently  has  2  procedures  to  share  courses:  either  1)  the  export  of  a  course  where 
it  is  presumed  that  the  receiving  GIFT  user  has  a  full  installation  GIFT  software 
and  third-party  applications  or  2)  to  export  a  course  with  an  entire  install  of  GIFT, 
resulting  in  a  relatively  large  software  package  (500  MB).  There  are  2  concerns  in 
this  area  that  require  further  attention:  1)  examination  of  the  export  package  for 
more  efficient  distribution  and  2)  how  to  handle  configuration  errors  when  the 
receiving  user  is  missing  required  hardware  or  software  components.  These  issues 
are  applicable  even  within  a  cloud-based  implementation  of  GIFT. 
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7.  Interdependencies  with  Other  Adaptive  Training  Research 
Vectors 


This  section  examines  interdependencies  between  authoring  and  the  other  5 
adaptive  training  research  vectors  (Fig.  2).  This  discussion  forms  the  basis  for  the 
sequencing  of  research  and  ultimately  bringing  adaptive  training  capabilities  into 
a  state  of  practice. 
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Fig.  2  Adaptive  training  research  vectors 

Authoring  tools  and  technologies  enable  the  selection  and  configuration  of  learner 
models,  instructional  pedagogy,  domain  content,  and  evaluation  metrics, 
respectively.  Conversely,  the  products  and  insights  generated  from  each  of  the 
other  vectors  provide  guidance  to  the  requirements  for  the  authoring  experience. 
Finally,  reusable  content  and  automated  authoring  rely  upon  architecture  to  fully 
realize  those  capabilities. 

7.1  Learner  Modeling  and  Authoring 

With  respect  to  authoring,  tutor  creators  will  require  guidance  and  tools  to  select 
and  configure  the  appropriate  dimensions  of  a  learner  model  for  a  given  course. 
This  might  require  the  content  creator  to  select  a  subset  of  data  points  from  a 
larger  long-term  learner  record  and/or  integrate  new  attributes  into  the  model. 
Likewise,  authors  will  need  to  be  able  to  specify  where  and  how  these  data  are 
used  within  the  tutor,  as  well  as  how  to  add  the  data  back  into  a  long-tenn  learner 
record  store  upon  completion  of  the  course.  Accurate  modeling  of  the  learner  is 
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critical  to  driving  instructional  decisions  in  adaptive  training  systems,  but  research 
is  needed  to  determine  what  this  dataset  will  look  like.  Candidates  abound  in  the 
literature,  but  in  general,  these  include  transient  data/states,  cumulative  states 
(building  over  time),  and  enduring  data/states  (Paneva  2006).  Transient  measures 
of  importance  include  individual  behavioral  and  physiological  data,  and  cognitive, 
affective,  physical,  and  social  states  to  represent  learning.  Cumulative  measures 
include  achievements  (e.g.,  certifications,  training,  education,  and  experiences), 
affiliations,  work  history,  and  domain  competency.  More  enduring  infonnation 
about  the  learner  might  include  gender,  culture,  first  language,  physical 
constraints  (e.g.,  colorblind/deaf),  values,  personality  attributes,  or  other  trait- 
based  infonnation  (Sottilare  and  Brawner  2014).  All  of  these  measures/states  are 
potential  drivers  for  adaptive  training  decisions.  Authoring  support  for  learner 
modeling  may  take  the  fonn  of  useful  user  interfaces,  semi-automated 
configuration  support,  and  connection  gateways  to  external  learner  database 
systems.  Philosophically,  an  author  should  have  to  author  items  related  to  the 
assessment  of  student  knowledge  and  experiences,  but  the  individual  differences, 
states,  and  trait  infonnation  should  be  abstracted  away  from  their  use  in 
nonresearch  settings. 

7.2  Automated  Instruction  and  Authoring 

Authoring  will  work  with  the  instructional  management  vector  to  outline  the 
author’s  role  in  instructional  management.  Specifically,  we  will  work  to 
detennine  what  level  of  control  is  appropriate  with  respect  to  authoring 
instructional  strategies.  We  will  receive  the  outputs  of  the  instructional 
management  vector  to  detennine  how  to  design  the  authoring  interaction  for  the 
management  and  selection  of  content  appropriate  to  a  chosen  instructional 
strategy  (e.g.,  MerrilTs  component  display  theory,  Gagne’s  9  events  of 
instruction).  The  instructional  management  and  authoring  vectors  will  also 
collaborate  on  larger,  cross-vector  issues,  including  the  design  of  adaptive 
tutoring  for  teams.  Generally,  the  goal  is  not  to  involve  automated  instruction  in 
the  authoring  process;  philosophically,  the  content  author  or  subject  matter  expert 
should  only  require  knowledge  of  a  domain,  while  the  system  should  know  best 
how  to  instruct  it.  If  any  authoring  decisions  are  involved,  they  should  be  limited 
to  a  specific  subset  of  available  models,  with  default  values  well  informed  from 
the  state  of  practice. 

7.3  Domain  Modeling  and  Authoring 

With  respect  to  domain  modeling,  authoring  knowledge  representations  for 
purposes  of  intelligent  tutoring  is  a  significant  challenge  within  the  current 
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iteration  of  the  GIFT  authoring  tools,  with  the  authoring  and  domain  modeling 
groups  being  tightly  linked.  We  will  exchange  data  and  information  with  the 
domain  modeling  vector  to  synchronize  author  requirements  for  domain 
modeling,  detennine  the  level  of  transparency  of  the  domain  modeling  process  to 
the  end  user,  and  develop  user  interfaces  for  modeling  knowledge,  feedback,  and 
other  instructional  interventions.  This  will  be  performed  with  an  emphasis  toward 
the  automation  of  items  that  can  be  automated,  or  with  an  emphasis  toward  mental 
models  where  the  author  must  be  involved. 

7.4  Evaluation  and  Authoring 

Training  effectiveness  is  concerned  with  the  impact  of  training  on  learning, 
retention,  and  transfer.  Outputs  of  training  effectiveness  research  will  specify 
methods  for  measuring,  analyzing,  and  instantiating  instructional  strategies  across 
different  training  domains  and  team  configurations.  In  practice,  these  measures 
may  take  the  form  of  data  hooks  that  will  be  built  into  the  tutor  to  facilitate 
automated  data  collection  for  either  real-time  or  posttraining  analysis.  There  is 
likely  to  be  considerable  overlap  between  the  data  needs  of  training  effectiveness 
and  those  of  the  tutor  itself.  Therefore,  there  exists  an  opportunity  to  take 
advantage  of  these  overlaps  in  the  specification  of  semi-automated  and  automated 
authoring  interfaces,  leading  to  a  more  robust  authoring  suite,  thus  reducing  the 
time  and  resources  required  to  develop  a  complete  tutoring  solution  for  a  given 
domain.  Data  hooks  for  the  automated  conduct  of  authoring  studies  should  be 
developed  and  used  in  the  same  fashion  as  many  of  the  web-based  services  now 
commercially  available.  While  not  expressly  a  research  task,  the  task  of 
developing  new  mental  models  to  support  new  types  of  system  authoring  is  a 
research  task  that  will  be  infonned  through  these  now-traditional  methods. 

7.5  Architecture  and  Authoring 

The  Architecture  vector  interacts  with  each  of  the  other  vectors,  including 
Authoring.  Architecture  and  Authoring  will  work  together  to  close  the  gap 
between  the  user  mental  model  and  the  system  conceptual  model — Authoring  will 
provide  guidelines  for  the  design  of  the  authoring  tool  interface  and  the  overall 
user  experience.  The  authoring  and  architecture  group  are  tightly  linked  in  the 
same  manner  as  the  authoring  and  domain  modeling  group;  one  relies  upon  the 
user  mental  model,  while  the  other  relies  upon  the  computerized  content  model. 
Architecture  will  specify  the  technologies  for  which  user  experiences  are 
required,  and  both  teams  will  work  together  to  bring  new  functionality  online, 
including  social  media  integration,  reusable  components,  and  content 
management  systems.  Finally,  it  is  expected  that  Architecture  will  enable  greater 
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automation  of  adaptive  tutoring  content  over  time,  which  will  require  parallel 
development  in  the  authoring  experience. 

8.  Conclusions 


This  report  outlines  ARL’s  plans  for  conducting  research  in  adaptive  training  and 
education  to  support  the  ALM  (US  Army  Training  and  Doctrine  Command  2011). 
Specifically,  this  report  relates  to  research  and  prototype  development  of 
authoring  tools  and  methods.  We  seek  to  the  answer  to  the  question,  What 
adaptive  training  methods  provide  the  best  value  (in  terms  of  effectiveness  and 
affordability)  for  the  authoring  of  US  Army  Training  and  Education  tutoring 
domains? 

This  report  outlined  the  following  research  goals: 

•  [For  human  authoring]:  Apply  usability  heuristics  to  develop  best 
practices  to  guide  authoring  of  adaptive  systems  based  on  the  author’s  role 
(e.g.,  domain  expert,  instructional  designer,  or  course  manager). 

•  [For  computer-aided  authoring]:  Identify  elements  of  the  authoring 
process  that  are  candidates  for  automated  or  semi-automated  authoring 
processes  to  remove  the  human  from  the  authoring  process  wherever 
possible. 

•  [For  leveraging  existing  training  and  educational  capabilities]:  Establish 
standards  for  the  integration  of  functionally  disparate  tools  and 
technologies  (e.g.,  external  training  systems  and  serious  games)  that  may 
be  relevant  to  adaptive  systems  to  reduce  the  need  for  authoring. 

These  research  goals,  in  turn,  support  the  following  goals  of  the  authoring 

vector: 

•  Decrease  the  resources  (time,  cost,  etc.)  required  to  author  an  intelligent 
tutor. 

•  Decrease  the  skill  threshold  required  by  various  user  groups  associated 
with  authoring  and  managing  an  intelligent  tutor. 

•  Enable  rapid  prototyping  of  intelligent  tutors  for  rapid  design  and 
evaluation  of  capabilities. 

•  Develop  standards,  including  common  tools  and  interfaces,  for  tutor 
authoring. 

•  Promote  reuse  of  content,  modules,  and  data  structures  in  tutors. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


AAR 

after-action  review 

AI 

artificial  intelligence 

ALM 

US  Army  Learning  Model 

ARL 

US  Army  Research  Laboratory 

CBT 

computer-based  training 

CBTS 

computer-based  tutoring  systems 

CMS 

Content  Management  System 

GAMETE 

Game-based  Architecture  for  Mentor-Enhanced  Training 
Environments 

GIFT 

Generalized  Intelligent  Framework  for  Tutoring 

ITS 

intelligent  tutoring  system 

MOS 

military  occupational  specialty 

RLO 

reusable  learning  object 

SRL 

self-regulated  learning 

WFO 

Warfighter  Outcome 

XML 

extensible  Markup  Language 
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